Patient video monitoring systems and methods having detection algorithm recovery from changes in illumination

ABSTRACT

Various embodiments concern video patient monitoring with detection zones. Various embodiments can comprise a camera, a user interface, and a computing system. The computing system can be configured to perform various steps based on reception of a frame from the camera, including: calculate a background luminance of the frame; monitor for a luminance change of a zone as compared to one or more previous frames, the luminance change indicative of patient motion in the zone; and compare the background luminance to an aggregate background luminance, the aggregate background luminance based on the plurality of frames. If the background luminance changed by more than a predetermined amount, then the aggregate background luminance can be set to the background luminance, luminance information of the previous frames can be disregarded, and motion detection can be disregarded.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 61/753,991, filed Jan. 18, 2013, which isincorporated herein by reference in its entirety.

FIELD OF INVENTION

The present disclosure relates generally to systems and methods forpatient monitoring by analyzing video frames to detect patient events.More particularly, the present disclosure relates to detectionalgorithms that can recover from unpredictable changes in illuminationof the patient area being monitored while avoiding false alarms andimproving detection of particular situations.

BACKGROUND

Healthcare facilities rely on patient monitoring to supplementinterventions and reduce the instances of patient falls. Constanteyes-on monitoring of patients can be difficult for healthcareprofessionals to maintain. Video monitoring can be used to automatepatient monitoring and increase the ability of a healthcare professionalto effectively monitor a group of patients distributed between differentrooms. Various systems and methods for patient video monitoring havebeen disclosed, such as U.S. Patent Application No. 2009/0278934entitled System and Method for Predicting Patient Falls, U.S. PatentApplication No. 2010/0134609 entitled System and Method for DocumentingPatient Procedures; U.S. Patent Application No. 2012/0026308 entitledSystem and Method for Using a Video Monitoring System to Prevent andManage Decubitus Ulcers in Patients, and U.S. Provisional PatentApplication No. 61/707,227 entitled System and Method for Monitoring aFall State of a Patient and Minimizing False Alarms.

Various routines can be run to analyze the output of a camera andidentify events. An alert can be issued to summon a healthcareprofessional to intervene when events are detected. Such an automatedsystem may be susceptible to light noise and other factors that cancause false alarms, which can burden a staff of healthcare professionalswith unnecessary interventions. Video monitoring can rely on lighting ofvarious kinds to allow visualization of the patient area and identifyparticular situations of interest. However, unpredictable changes inlighting of the patient's room can cause erroneous detections ofsituations warranting intervention and thereby causing false alarms.There exists a need for systems and methods for accounting forunpredictable changes in lighting to reduce the incidence of erroneousdetections and false alarms.

SUMMARY

Various embodiments concern a system for monitoring a patient in apatient area having one or more detection zones. The system can comprisea camera, a user interface, and a computing system configured to receivea chronological series of frames from the camera. The computing systemcan further be configured to perform the following steps based on thereception of each frame of the chronological series: calculate abackground luminance of the frame; calculate an aggregate backgroundluminance based on a respective background luminance for each aplurality of previous frames of the chronological series; and for eachof one or more zones, calculate a zone luminance based at least in parton the background luminance and a plurality of luminance values of aplurality of pixels of the zone. For each of one or more zones, thecomputing system can further be configured to detect patient motionbased on a change between the zone luminance and a previous zoneluminance of a previous frame exceeding a zone threshold. The computingsystem can further be configured to compare the background luminance ofthe frame to an aggregate background luminance and, if the backgroundluminance changed relative to the aggregate background luminance by morethan a threshold amount, disregard luminance information from theplurality of previous frames when performing the detecting patientmotion step and the calculating the aggregate background luminance stepfor subsequent frames of the chronological series. If patient motion isdetected, the computing system can generate an alert with the userinterface if the background luminance changed relative to the aggregatebackground luminance by less than the threshold amount but refrain fromgenerating the alert if the background luminance changed relative to theaggregate background luminance by more than the threshold amount.

The computing system can be configured to calculate the backgroundluminance of the frame by determining an average pixel luminance basedon each pixel of the frame. The computing system can configured tocalculate the aggregate background luminance by averaging the respectivebackground luminance for each the plurality of previous frames of thechronological series. The computing system can configured to calculatethe aggregate background luminance based on the background luminance ofthe frame and the respective background luminance for each the pluralityof previous frames of the chronological series.

In various embodiments, a computing system of a monitoring system can beconfigured to calculate a background luminance of a frame, and for eachzone of one or more zones of the frame, monitor for a luminance changeof the zone as compared to one or more frames of a plurality of framesthat were previously received from the camera, the luminance changeindicative of patient motion in the zone. The computing system canfurther be configured to compare the background luminance of the frameto an aggregate background luminance, the aggregate background luminancebased on the plurality of frames, and if the comparison determines thatthe background luminance changed relative to the aggregate backgroundluminance by more than a threshold amount, set the aggregate backgroundluminance to the background luminance of the frame and disregardluminance of the plurality of frames in subsequent monitoring for theluminance change in the one or more zones.

While multiple embodiments are disclosed, still other embodiments of thepresent invention will become apparent to those skilled in the art fromthe following detailed description, which shows and describesillustrative embodiments of the invention. Accordingly, the drawings anddetailed description are to be regarded as illustrative in nature andnot restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of a monitoring system.

FIG. 2 is a block diagram of components of a monitoring system.

FIG. 3 is a flowchart of a method for initializing a monitoring system.

FIG. 4 is a schematic diagram of a patient area.

FIG. 5 is a flow chart of a method for monitoring a patient andaccounting for changes in illumination.

FIG. 6 is another flow chart of a method for monitoring a patient andaccounting for environmental illumination changes.

While the subject matter of the present disclosure is amenable tovarious modifications and alternative forms, specific embodiments havebeen shown by way of example in the drawings and are described in detailbelow. The intention, however, is not to limit the invention to theparticular embodiments described. On the contrary, the invention isintended to cover all modifications, equivalents, and alternativesfalling within the scope of the invention as defined by the appendedclaims.

DETAILED DESCRIPTION

Various embodiments of the present disclosure concern video monitoringto detect patient events. Such events can concern situations in which apatient is at increased risk or otherwise is in need of intervention.Patient events can include a patient at risk of falling, a patientfalling, a patient outside of a designated area, and patient motion,among various other events.

FIG. 1 is a schematic diagram of a patient monitoring system 10. Thepatient monitoring system 10 can allow a healthcare professional tomonitor multiple patient areas 12-15 from a monitoring station 11 via acomputing system 17. The monitoring station 11 can comprise a userinterface, which can include a screen and an input. The screen candisplay images of the patient areas 12-15, indications of one or morestates of the patients being monitored, patient data, and/or otherinformation. In some embodiments, the components of the monitoringstation 11 are portable such that the monitoring station 11 can movewith the healthcare processional.

While four patient areas 12-15 are shown in FIG. 1, any number ofpatient areas can be monitored at the monitoring station 11 via thecomputing system 17. The monitoring station 11 can be remote from thepatient areas 12-15. For example, the monitoring station 11 can be onthe same or different floor as the patient area 12-15, in the same ordifferent building as the patient area 12-15, or located in ageographically different location as the patient area 12-15.Furthermore, the patient areas 12-15 can be remote from each other. Thecomputing system 17 can be in one particular location or the componentsof the computing system 17 can be distributed amongst multiplelocations. The computing system 17 can be at the monitoring station 11or can be remote from the monitoring station 11 and/or the patient areas12-15.

As shown in FIG. 1, a plurality of cameras 18-21 can be respectivelypositioned to view and generate frames of the plurality of patient areas12-15. Information concerning the frames, such as analog or digitalencodings of the frames, can be transmitted from the plurality ofcameras 18-21 along data channels 16 to the computing system 17. In somecases, the computing system 17 is a single unit, such as a server or apersonal computer (e.g., a desktop computer or a laptop computer). Insome cases, the computing system 17 is distributed amongst severalunits, such as one or more personal computers, one or more servers,circuitry within one or more of the cameras 18-21, and/or othercomputing devices. In some cases, the computing system 17 is part of acloud computing network. The data channels 16 can be wired lines of anetwork (e.g., a local area network) and/or wireless channels (e.g.,Wi-Fi or cellular network).

Each of the plurality of cameras 18-21 can generate a chronologicalseries of frames (e.g., as images). The plurality of cameras 18-21 canbe analog or digital cameras. Each of the plurality of cameras 18-21 cancapture a sequence of frames at a predetermined frame rate, such as six,eight, sixteen, twenty-four, or some other number of frames per second.The resolution of digital cameras is usually defined by the number ofpixels both horizontally and vertically (such as 640×480) or as a totalnumber of pixels in the image (such as 1.4 mega pixels), while theresolution of analog video cameras is typically defined by the number oftelevision lines. Analog frames can be converted to digital frames byanalog-to-digital conversion circuitry (e.g., as part of the computingsystem 17 and/or the plurality of cameras 18-21). The plurality ofcameras 18-21 can have infrared illumination or night visioncapabilities for operating in low light conditions.

FIG. 2 shows a block diagram of circuitry of the monitoring system 10.Although camera 18 is specifically shown as an exemplar, the componentsof the camera 18 can be included as part of each camera and themonitoring system 10. The camera 18 can include optics 30. Optics 30 caninclude a lens, a filter, and/or other components for capturing andconditioning the light of the patient area. The camera 18 can furtherinclude a sensor 31 for converting light from the optics 30 intoelectronic signals. Different types of sensors 31 can be used dependingon whether the camera 18 is analog (e.g., generating analog video) ordigital (e.g., generating discrete digital frames). The sensor 31 caninclude a charge-coupled device (CCD) or a complementarymetal-oxide-semiconductor (CMOS).

The camera 18 can further include a processor 32 and memory 33. Theprocessor 32 can perform various computing functions, such as thosedescribed herein or otherwise useful for operating the camera 18. Thememory 33 can be a non-transient computer readable storage medium (e.g.,random access memory or flash) for storing program instructions and/orframes. For example, the processor 32 can be configured to executeprogram instructions stored on the memory 33 for controlling the camera18 in converting light from the patient area 12 into digital signalswith the sensor 31, storing the digital signals on the memory 33 asframe data, transferring the frame data to the computing system 17,and/or performing any other function. The processor 32 may performvarious signal conditioning and/or image processing on the frames. Theprocessor 32 may include a dedicated video processor for imageprocessing. Although not illustrated, the camera 18 can further includea network interface controller and a power supply. The camera 18 mayinclude a user interface which can include user controls and/or anaudible alarm.

The computing system 17 can comprise a single housing or multiplehousings among which circuitry can be distributed. The computing system17 can include display circuitry 34 which can provide a graphics outputto a screen. Display circuitry 34 can include a graphics processor andgraphics memory which can support user interface functionality. Displaycircuitry 34 may be part of a separate display, such as a screen,handheld device, or remote terminal. Display circuitry 34 can facilitatethe display of frames taken by the camera 18 of the patient area 12 on ascreen and/or patient status information. User input circuitry 35 caninclude components for accepting user commands such as a keyboard,mouse, trackball, touchpad, touch screen, joystick, slider bar, or anyother control. User input circuitry 35 can facilitate the definition ofboundaries and monitoring zones, as will be further described herein.

The computing system 17 can include a processor 36 and memory 37. Thememory 37 can be one or more discrete non-transient computer readablestorage medium components (e.g., RAM, ROM, NVRAM, EEPROM, and/or FLASHmemory) for storing program instructions and/or data. The processor 36can be configured to execute program instructions stored on the memory37 to control in the computing system 17 in carrying out the functionsreferenced herein. The processor 36 can comprise multiple discreteprocessing components to carry out the functions described herein as theprocessor 36 is not limited to a single processing component. Thecomputing system 17 can include a network controller 38 for facilitatingcommunication with the cameras 18-21 and/or other remote components. Thecomputing system 17 can include a power supply 39 which can facilitate aconnection to an electrical outlet and/or the power supply 39 cancomprise a battery. Whether distributed or unified, the components ofthe computing system 17 can be electrically connected to coordinate andshare resources to carry out functions.

FIG. 3 illustrates a flow chart of a method for setting up a monitoringsystem to monitor a patient. The method includes receiving 50 a frame ofa patient area. The frame can be an image generated by the camera 18 andcan be received 50 by the computing system 17 of FIGS. 1-2. The framecan be displayed on a screen at the monitoring station 11. The patientarea can correspond to any area associated with a patient and/or whereincreased risks to the patient are anticipated. Such areas can include abed, a chair, a wheelchair, a tub, a shower, and/or an entryway.

The method can further include receiving 51 an input indicating alocation. The input may be provided by a healthcare professional at themonitoring station 11 with a user interface. The indicated location cancorrespond to an area occupied by the patient and/or a risk to thepatient. In some cases, a healthcare professional can indicate one ormore boundaries associated with heightened fall risk, such as the leftand right sides of a bed. For example, the healthcare professional candesignate the left side and the right side of the bed 60 shown in FIG.4. It is noted that patient areas may be automatically identifiedthrough a pattern recognition procedure implemented in the monitoringsystem in some embodiments.

One or more zones can be defined 52 based on the received 51 user input.For example, one or more zones within and/or outside of a boundarydesignation can be defined by an algorithm. The one or more zones cancorrespond with areas of increased risk to a patient, such as areasadjacent to and/or over the side of the bed (e.g., as a virtualbedrail). In some embodiments, the one or more zones correspond to areaswhere patient motion is expected and/or where patient motion can beindicative of dangerous or otherwise unwanted activity. In some cases, aplurality of zones can be defined 52 to cover the bed and/or extendalong the bed or other patient area. Various rules can be used by analgorithm of a monitoring system to define 52 the one or more zones. Forexample, a longitudinal dimension of a zone can be defined 52 asextending parallel with the longitudinal dimension of a bed as indicatedby the received 51 input. The zones can be defined 52 in dimensions ofpixels. A width dimension of a zone can be defined as extending outwardfrom the boundary designations (e.g., away from the bed) for apredetermined number of pixels. The predetermined number of pixels maycorrespond with a distance, such as twelve inches. FIG. 4 shows fivezones that can be manually or automatically defined 52 based on userinput. Specifically, a left outer zone 61, a left inner zone 62, acenter zone 63, and right inner zone 64, and a right outer zone 65 aredefined 52 along the bed 60.

The method of FIG. 3 further includes monitoring 53 the defined 52zones. Motion in the center zone 63 may be indicative of normal patientmovement, but motion detected within either of the left inner zone 62 orthe right inner zone 64 can indicate that the patient is at leastpartially off the bed 60 and at risk of falling. Different rules formonitoring 53 can be applied for the inner zones 62 and 64 as comparedto the outer zones 61 and 65. For example, the inner zones 62 and 64,where the patient is most likely to fall, can be monitored 53 to triggera fall alert or other indication. The outer zones 61 and 65, which arepositioned to intercept visitors, can be monitored 53 to temporarilydisarm the monitoring system to prevent a visitor from erroneouslytriggering the fall alert or other indication.

Monitoring 53 of the zones can allow particular discriminationalgorithms to be run using the pixels of the zones to identify patternsor other indication of a patient event. For example, an algorithm cancompare pixel characteristics of a zone between sequential frames todetermine whether the pixel characteristic of the zone changed, thechange indicative of patient movement. While monitoring 53 algorithmswill be further described herein, it may be useful to describe pixelluminance and other pixel characteristics used in monitoring 53.

Each pixel of a frame can be associated with luminance and other opticalcharacteristics. The luminance characterizes the intensity of the lightassociated with the pixel. Luminance can be particularly useful becauseluminance can be measured in dark environments, such as at night.Luminance can be used for motion detection by identifying changes in theluminance of a zone over time. The luminance of a zone may change overtime (e.g., between sequential frames) because the reflectance of thesurfaces within the zone can change due to movement of the surfaces. Forexample, a patient's arm can move into a zone, thereby changing theluminance of the zone to be darker or lighter. Comparisons between someluminance-based metric of a zone between frames can determine whetherthe change in luminance is indicative of patient movement within thezone. For example, the change in luminance may be compared to athreshold, the threshold distinguishing small changes in luminanceunlikely to be associated with patient movement (e.g., noise) and largerchanges in luminance likely to be from patient movement. However,changes in illumination of the patient area can cause larger changes inluminance of the zone. Such a large change in luminance in the zone canbe misidentified as patient motion or other event, potentiallytriggering a false alarm.

Referring to FIG. 4, the patient area 12 includes several sources oflight that can change the luminance of the zones 61-65 and frustrateaccurate detection of patient motion and/or cause erroneous detections.For example, turning the television 66 on and off can change theillumination of the patient area 12. Furthermore, the television 66 cancause light flicker within the patient area 12 as the screen changesbetween bright and dark scenes, which is referred to herein as flickernoise. The lamp 67 or other lighting device can be turned on and off tochange the illumination of the patient area 12. The window 68 can allowoutside light into the room. The amount of outside light coming throughthe window 68 can change due to clouds unpredictably blocking andunblocking direct sunlight. An outside light can come through thedoorway 69 to change the illumination of the room. The outside light canbe turned on and off and/or the doorway 69 can be obscured by the dooropening/closing or a person moving through the doorway 69. A cameratransitioning between day and night modes (e.g., between ambient lightand infrared operation) can abruptly change the perceived illuminationof the patient area 12. Such changes in illumination can cause changesin the luminance of pixels within the zones 61-65 and cause erroneousdetections of motion and false alarms. Various embodiments of thepresent disclosure are directed to systems and methods for accountingfor such changes in illumination to limit erroneous detections and falsealarms. Such methods are further described in connection with FIGS. 5and 6.

FIG. 5 shows a flow chart of a method for monitoring a patient whileaccounting for changes in illumination of the patient area. The methodcan be implemented in a monitoring system (e.g., as programinstructions) as referenced herein. Some embodiments of the method maybe preceded by an initial setup procedure, such as that of FIG. 3. Inany case, the method of FIG. 5 includes the reception 70 of a frame. Theframe can be part of a chronological series of frames generated by thecamera and transmitted to a computing system in sequence, wherein thesteps of the method are performed for the reception 70 of eachrespective frame of the chronological series. Accordingly, the methodsteps of FIG. 5 represent one iteration of a cycle, the cycle beingrepeated for each frame received 70.

The method includes comparing 71 background luminance of the received 70frame to background luminance of a plurality of previously received 70frames. The background luminance of a frame can be based on an aggregatemeasure of the respective luminance values of most or all of the pixelsof the frame. As such, the background luminance can represent thegeneral illumination of the patient area and the overall luminance ofthe frame. In some cases, the background luminance can be calculatedonly from pixels of a specific selection of the patient area that is notoccupied by the patient or otherwise part of a zone monitored formovement (e.g., an area of the floor and/or wall). While the backgroundluminance can be calculated according to various techniques, somespecific techniques for calculating background luminance are elsewherediscussed in the present disclosure.

The comparison 71 of the background luminance of the frame to thebackground luminance of the plurality of previous frames can includedetermining a difference between the background luminance of the currentframe (i.e. the frame received 70 in the current iteration of the methodcycle) and the background luminance of a plurality of frames previouslyreceived 70 in previous cycles (e.g., five frames of the five mostrecently completed cycles). The background luminance of the plurality offrames can be aggregated into a single value, such as an aggregatebackground luminance of the plurality of frames, to which the backgroundluminance of the current frame can be compared 71. For example, theaggregate background luminance of the plurality of frames can be theaverage of the background luminance of each of the plurality of previousframes. The use of the plurality of previous frames can function as amoving average of the background luminance over time, to which thebackground luminance of each current frame is compared 71. Thecomparison 71 can include determining a difference between thebackground luminance of the frame and the background luminance of theplurality of previous frames. The difference can be determined bysubtraction and then optionally taking the absolute value of the result.While the comparing 71 step states that the background luminance of theframe is compared 71 to the background luminance of a plurality ofprevious frames, in some other embodiments the comparing 71 stepsconcern comparing 71 the background luminance of the frame to backgroundluminance of one of the plurality of previous frames.

Based on the comparison 71, the method can include determining whetherthe difference in background luminance between the current frame and thebackground luminance of the plurality of previous frames exceeds 72 abackground threshold. The background threshold can be set to distinguishbetween changes in background luminance due to motion and larger changesin background illumination due to a change in the illumination of theroom (e.g., due to television noise flicker, a light, and/or the sun).The background threshold can be set along any luminance-based metric.The background threshold can be a threshold amount. The threshold amountcan be a predetermined amount (e.g., set by a user or a factorysetting). A user can set the threshold based on a risk assessment. Aparticular example value for the background threshold is discussedelsewhere herein or based on a history of false detections. Thethreshold amount can be a dynamic threshold amount. For example, thedynamic threshold amount can be based on the current backgroundluminance (e.g., 10% of current background luminance), such that thethreshold is smaller in darker environments and larger in brighterenvironments. The dynamic threshold amount can be based on a statisticof the background luminance, such as a percentage of the standarddeviation of the luminance of the pixels of the current frame or aplurality of previous frames. The dynamic threshold can be based on therelative amount of infrared content and non-infrared content of thebackground luminance. In some cases, multiple changes in backgroundluminance over a plurality of frames that do not exceed the thresholdcan cause the threshold to be automatically raised

If the difference in background luminance does not exceed 72 thebackground threshold, then the method can continue with monitoring 73for a luminance change in one or more zones of the frame as compared toone or more previous frames. As discussed herein, motion and otherevents can be detected by comparing some luminance-based metric of aparticular zone between a current and one or more prior frames todetermine whether the value of the metric changed. Accordingly, themethod can include determining whether a change in luminance of a zoneexceeds 75 a zone threshold. The zone threshold can represent thedifference in luminance between insubstantial changes due to low levelsof noise and larger changes due to motion or other important events.

While changes in luminance can be identified by directly comparing zoneluminance values from consecutive frames, changes in luminance can bedetermined by additional or alternative techniques. For example, anotherluminance-based metric is the number of edges within a zone. Edges canbe detected by analyzing the contrast in luminance between neighboringpixels. High contrast indicates an edge while low contrast indicates thelack of an edge. Summing the edges detected within a zone can be usefulin motion detection because the number of edges within a zone changefrom one frame to the next if motion is occurring within the zone. Assuch, monitoring 73 for a luminance change can include determiningwhether a change in the number of edges detected in a zone between twoframes exceeds 75 a zone threshold.

If the zone threshold is exceeded 75, then an indication can begenerated 76, such as a fall alarm and/or a changing of a patient state.The cycle can then be repeated with the reception 70 of the next frameof the chronological series. However, returning to the step ofdetermining whether the difference in background luminance between thecurrent frame and the background luminance of the plurality of previousframes exceeds 72 the background threshold, various steps can be takenif the background threshold is exceeded 72. In some cases, opticalcharacteristic information (e.g., luminance values, color information,chrominance) of the plurality of previous frames can be disregarded 74in subsequent iterations of the method cycle if the background thresholdis exceeded 72. Disregarding 74 can include erasing luminance and/orother information from memory. In some embodiments, disregarding 74 mayinclude not referencing the optical characteristic information of theplurality of previous frames in subsequent iterations of the cycle.Luminance values and/or other information of the plurality of previousframes may otherwise be referenced in the comparison 71 step and themonitoring 73 step in subsequent iterations of the cycle, yet thisinformation will be disregarded 74 because the background thresholdbeing exceeded 72 indicates that subsequent changes in luminance arelikely due to the changes in environmental illumination and not motionwithin a zone. Accordingly, the optical information of the previousframes can be disregarded 74 so that the steps of the method that relyon the optical information of one or more previous frames (e.g.,luminance background comparison 71 and zone monitoring 73) can refer tooptical information that is more likely to reflect the currentillumination conditions of the patient area. In some embodiments, alloptical information of the plurality of previous frames can bedisregarded 74. In some other embodiments, only some of the opticalinformation of the plurality of previous frames may be disregarded 74,such as luminance, chrominance, and/or edge counts.

If the background threshold is exceeded 72, subsequent cycles of themethod can refer to the background and zone luminance of the currentframe (i.e. the frame for which the background threshold being exceeded72 was detected), but not luminance values of frames preceding thecurrent frame. In the subsequent cycles, the current frame will become aprevious frame and may be referenced as a previous frame (e.g., in thecomparison 71 and monitoring steps 73). Moreover, more framessubsequently received 70 can be referenced as previous frames forcomparing past and current luminance until the background threshold isagain exceeded 72, at which point the luminance values and/or otheroptical information of the previous frames are disregarded 74. Inembodiments where the zone threshold represents a change in the numberof edges in a zone, or detection of motion is otherwise based on thenumber of edges in a zone, the edge information can be disregarded 74 ifthe background threshold is exceeded 72. Other types of information usedto detect motion can be disregarded 74 for the plurality of previousframes if the background threshold is exceeded 72.

The current frame, as the term is used herein, refers to the frame mostrecently received and for which the computing system is processing todetect motion, a change in background luminance, and/or another event. Aprevious frame, as the term is used herein, refers to a frame that wasreceived previous to the current frame and represents a time thatprecedes the time represented by the current frame. The current framecan become a previous frame when the next frame (i.e. the new currentframe) is received and monitoring based on the next frame begins, wherethe next frame becomes the current frame. If luminance information ofthe plurality of previous frames is disregarded 74, the luminanceinformation of the current frame is not likewise disregarded 74 eventhough the current frame becomes a previous frame in the next cycle.Rather, disregarding 74 the luminance information of the plurality ofprevious frames refers to those frames which were previous frames at thetime of the disregarding 74 step.

FIG. 6 shows a flow chart of a method for monitoring a patient whileaccounting for changes in illumination of the patient area. Someembodiments of the method of FIG. 6 can also correspond to the flowchartof the method of FIG. 5 and/or other embodiments referenced herein. Themethod of FIG. 6 can be implemented in a monitoring system (e.g., asprogram instructions) as referenced herein. It is noted that theflowchart of FIG. 6 contains three subroutines that can each beperformed as different parts of a cycle, each iteration of the cyclebased on the reception 81 of a frame from a camera. Specifically, themethod includes a background luminance subroutine 80 that determines atleast one measure of the background luminance of the frame. The methodfurther includes a zone detection subroutine 90 that detects movement orother conditions within one or more zones defined in the frame. Themethod further includes an environmental illumination correctionsubroutine 100. The subroutines can run concurrently or consecutively invarious orders, however the background luminance subroutine 80 istypically completed before some or all of the steps of the other twosubroutines are started as the background luminance subroutine 80provides information to the other two subroutines.

The background luminance subroutine 80 includes receiving 81 a frame,which can occur in any manner referenced herein (e.g., the framegenerated by a camera and part of a chronological series of frames). Thebackground luminance subroutine 80 further includes determining 82 theluminance of each pixel of the frame. Luminance can be measured invarious different ways. One technique for measuring luminance includescategorizing the intensity of each pixel along an 8-bit grayscale, where0 can be the lowest possible intensity level (i.e. darkest) and 255 canbe the highest possible intensity level (i.e. brightest). While otherschemes for assessing the relative level of luminance of pixels arepossible, this technique will be used herein as an exemplar.

The background luminance subroutine 80 further includes calculating 83 abackground luminance value by averaging the luminance values of allpixels of the frame. The background luminance value is a measure of theluminance of the general pixels of the frame. Although the flowchart ofFIG. 6 specifically identifies averaging of pixel luminance to determinethe background luminance, various other aggregating techniques can beused to calculate a value representing the background luminance, such asmedian, mode, or standard deviation, among others.

The background luminance subroutine 80 can further include determining84 an aggregate background luminance from historical backgroundluminance. The historical background luminance can represent thebackground luminance or one or more previous frames, such as the fivemost recent previous frames. The aggregate background luminance can be ameasure of the environmental luminance of multiple frames over time. Insome cases, the aggregate background luminance is determined 84 from thebackground luminance of the current frame and a plurality of previousframes, the plurality of previous frames corresponding to the historicalbackground luminance. The background luminance values of the pluralityof previous frames and optionally the background luminance of thecurrent frame can be averaged to determine 84 the aggregate backgroundluminance. Other aggregating techniques can be used to represent thebackground luminance values over a plurality of frames, such ascalculating the median, mode, or standard deviation of the backgroundluminance values. Use of the aggregate background luminance instead ofjust a current background luminance can help smooth out the backgroundluminance values and minimize transient changes in background luminance.However, it is noted that in some alternative versions of the method ofFIG. 6, the aggregate background luminance may not be determined 84 andinstead the background luminance is used in place for the subsequentsteps of the method. Use of the aggregate background luminance (oralternatively the background luminance) is further described inconnection with the zone detection subroutine 90 and the environmentalillumination correction subroutine 100.

The zone detection subroutine 90 includes, for each pixel in any zone,calculating 91 a difference in luminance between the luminance of thepixel and the aggregate background luminance. Alternatively, the stepmay comprise calculating 91 a difference in luminance between theluminance of the pixel and the background luminance of the currentframe. As discussed previously, each frame can have one or more zoneswhich are monitored for motion or other event. The zones can be those ofFIG. 4, for example. The difference in luminance can be calculated 91 bysubtracting the luminance of the pixel from the aggregate backgroundluminance or subtracting the aggregate background luminance from theluminance of the pixel, for example. In any case, the absolute value canbe the taken in calculating 91 the difference between the luminance ofthe pixel and the aggregate background luminance such that positive andnegative differences can be equally valued. For example, the aggregatebackground luminance may be 150 on the 8-bit grayscale, while a firstpixel has a luminance of 140 and a second pixel has a luminance of 165.The calculated 91 difference associated with the first pixel can be 10(assuming the absolute value is taken) and the calculated 91 differenceassociated with the second pixel can be 15.

The zone detection subroutine 90 can further include, for each zone ofthe frame, calculating 92 a zone luminance by aggregating thedifferences in luminance of the pixels of the zone. The zone luminancecan represent a measure of the overall luminance of a zone. A pluralityof zone luminance values can be calculated 92 for a plurality of zonessuch that each zone has a corresponding zone luminance. The aggregationcan include determining the sum, average, median, mode, standarddeviation, or other statistical metric from the calculated 91 differenceof the luminance of the pixels of the zone with the aggregate backgroundluminance. In the case of summing, and continuing with the exampleprovided above, the first and the second pixels can be in the same zone,and as such can be summed to 25. Further difference values from the samezone can be aggregated to calculate the zone luminance.

The zone detection subroutine 90 can further include, for each zone ofthe frame, calculating 93 a luminance difference between the calculated92 zone luminance of the zone and a historical zone luminance. Thehistorical zone luminance can be the zone luminance of the same zonecalculated 92 from the previous frame (e.g., calculated 92 in the mostrecently completed cycle of the method). The historical zone luminancecan be based on the zone luminance of the same zone over a plurality ofprevious frames (e.g., the zone luminance values of the previous fiveframes averaged together). The calculated 93 zone luminance differencecan represent the change in luminance of the zone between the previousframe and the current frame, which can be indicative of patient movementwithin the zone. As discussed previously, the calculated 93 zonedifference can concern a change in the number of edges identified in azone. The zone luminance difference can be compared to a zone thresholdto determine whether the zone luminance difference exceeds the zonethreshold. The zone threshold can represent the difference between minorchanges in luminance due to noise and larger changes in luminance due tomovement. The zone threshold can be set for each zone and may be thesame or different for the different zones. The zone threshold may be apredetermined amount. For example, the zone threshold may be 2 on the8-bit grayscale previously described.

If the zone threshold is exceeded 94, then the zone detection subroutine90 can generate 95 an indication with a user interface. The indicationmay comprise designating the zone as active, representing a notificationof an event on a screen, raising a patient fall state risk, and/orissuing an alert (e.g., an alarm to summon intervention). The zonedetection subroutine 90 can end 96 with the generation 95 of theindication or if the zone threshold is not exceeded 94. It is noted thatthe generation 95 of the indication of the patient risk may be suspendedby the environmental illumination correction subroutine 100 as furtherdiscussed herein.

The environmental illumination correction subroutine 100 can account forchanges in illumination between frames. In doing so, the environmentalillumination correction subroutine 100 can prevent erroneous eventdetections and false alarms by disregarding detections possibly causedby the change in illumination and can further reset various detectionvalues to facilitate the quick recovery of the detection algorithm forsubsequent cycles.

The environmental illumination correction subroutine 100 can includecalculating 101 a background luminance change based on the backgroundluminance of the current frame and the aggregate background luminance.The change can be a difference between the background luminance of thecurrent frame and the aggregate background luminance. The change can becalculated 101 by subtracting the background luminance of the currentframe from the aggregate background luminance or subtracting theaggregate background luminance from the background luminance of thecurrent frame, for example. In any case, the absolute value can be thetaken in calculating 101 the change such that a comparable differencecan be assured. The calculated 101 difference can be compared to abackground threshold to determine whether the background luminancechange exceeds 102 the background threshold. The background thresholdcan represent the difference between changes in illumination due topatient motion and larger changes in illumination likely to triggererroneous patient event detection. Continuing with the above example,the background threshold may be 10 on the 8-bit grayscale previouslydescribed. It is noted that the inventors of the subject matter of thisdisclosure have discovered that a threshold of about 10 on an 8-bitgrayscale can be particularly effective, in some implementations, atdistinguishing between luminance changes associated with patient motionand luminance changes associated with changes in environmentalillumination.

Several steps can be taken in response to the background luminancechange exceeding 102 the background threshold. In some cases, thegeneration 95 of indications based on the zone threshold being exceeded94 in the current cycle can be suspended 103. In some cases, the zonedetection subroutine 90 may be automatically ended 96 (regardless ofwhether the zone threshold is or will be exceeded 94) for the currentcycle if the background threshold is exceeded 102. It may be appropriateto disregard the zone threshold being exceeded 94 because detections ofmotion or other events may be the result of the environmentalillumination change and not actual motion. If the patient is indeedmoving, such motion can still be detected in the next cycle as thealgorithm recovers from the environmental illumination change.

The environmental illumination correction subroutine 100 can includesetting 104 the historical background luminance to the backgroundluminance of the current frame. As discussed previously, the aggregatebackground luminance can be determined 84 from the historical backgroundluminance, which itself can be based on the background luminance of aplurality of previous frames. However, the luminance of the previousframes may deviate sharply from the luminance of the current frame andsome number of subsequent frames if the environmental illumination ofthe patient area has changed. Accordingly, the method can includesetting the historical background luminance to the background luminanceof the current frame. It is noted that the historical backgroundluminance may preferably be calculated from a plurality of consecutiveframes (e.g., three or five frames) to provide a smoothing function,however it may be more important to forgo such smoothing when aenvironmental change in illumination is detected. Accordingly, if thehistorical background luminance is set 104 to a single frame (i.e. thecurrent frame) instead of being based on the plurality of previousframes, then in the next cycle the aggregate background luminance may bebased on the single frame or two frames (e.g., if the aggregatebackground luminance is determined 84 based on the background luminanceof the current frame and the historical background luminance). Thenumber of frames used to calculate the historical background luminancecan be built back up over several cycles to restore the smoothingfunction as the background luminance is calculated 83 for each framereceived 81.

For each zone, the historical zone luminance associated with the zonecan be set 105 to the zone luminance of the current frame. The zoneluminance can be calculated 92 as part of the zone detection subroutine90 or can be calculated separately (e.g., in case the zone detectionsubroutine 90 is not completed due to the background threshold beingexceeded 102). Setting 105 the historical zone luminance of each zone toa respective zone luminance of the current frame can help the zonedetection subroutine 90 to quickly adjust to the new illumination of thepatient area in the next cycle. In some cases, the historical zoneluminance of each zone may be based on a plurality of previous frames(e.g., a moving zone luminance average of consecutive frames), but ifthe environmental illumination of the patient area has changed, then thezone luminance of the previous frames are likely to be misleading fordetecting changes in zone luminance due to motion. Accordingly, themethod can include setting 105 the zone luminance for each zone to arespective zone luminance of the current frame that reflects theenvironmental change in illumination. The zone detection subroutine 90of the following cycle will then have a more relevant historical zoneluminance value with which to calculate 93 the zone luminance differencein determining whether a change in luminance of a zone is indicative ofmotion or other event. Although each of the suspending 103, setting 104the historical background luminance, and setting 105 the historical zoneluminance are each shown in the environmental illumination correctionsubroutine 100, various embodiments can selectively include or omitthese and other steps triggered in response to the background thresholdbeing exceeded 102. A new cycle may begin when the environmentalillumination correction subroutine 100 ends 106 and the next frame isreceived 81.

In addition to, or as an alternative to, setting 105 the historical zoneluminance, zone luminance information of a plurality of previous framescan be disregarded or otherwise dampened based on the backgroundthreshold being exceeded 102. The luminance information may concerncalculated 92 zone luminance values for each zone over the plurality ofprevious frames. Dampening luminance information is discussed elsewhereherein, and can include erasing the luminance information for the zonesof the plurality of previous frames or applying a scalar (e.g., 0.8) tothe data. Accordingly, in some cases, edge detection information fromeach of the zones can be erased or otherwise disregarded if thebackground threshold is exceeded 102.

The flowchart and block diagrams in the FIGS. of the present disclosureillustrate the architecture, functionality, and operation of somepossible implementations of systems, methods, and computer programproducts according to various embodiments of the present disclosure. Inthis regard, each step in the flowchart or arrangement of blocks mayrepresent a component, module, segment, or portion of code, whichcomprises one or more executable instructions for implementing thespecified logical function(s). It should also be noted that, in somealternative implementations, the functions noted in the steps may occurout of the order noted in the FIGS. or as otherwise described. Forexample, two steps shown in succession may, in fact, be executedsubstantially concurrently, or the steps may sometimes be executed inthe reverse order, depending upon the functionality involved.

It is noted that reception of a frame (e.g., by a computing system froma camera) does not necessarily mean reception of all of the data of theframe sufficient to reconstruct the entire frame. Rather, reception ofthe frame can include reception of representative data (e.g., luminanceinformation) that allows for calculation of the background luminance andother values for performing the functions described herein.

While luminance has been used as an example optical characteristicherein for adjusting a detection algorithm, various other opticalcharacteristics can be used in place of, or in addition to, luminance inany of the embodiments referenced herein. An optical characteristic canbe measured based on the intensity or degree of content. An opticalcharacteristic can include chrominance. Optical characteristics caninclude color content, or one or more particular components of color(e.g., red, green, blue, and/or other color). Color can be measured byany measure of color space. The term “luminance” can be replaced by“optical characteristic” in the method of FIG. 5 (or another anyembodiment specifically directed to luminance). For example, variousembodiments can concern a system for monitoring a patient in a patientarea having one or more detection zones, the system comprising: acomputing system configured to receive a chronological series of framesand perform the following steps based on the reception of each frame ofthe chronological series: calculate a background optical characteristicof the frame; for each zone of one or more zones of the frame, monitorfor a optical characteristic change of the zone as compared to one ormore frames of a plurality of frames that were previously received fromthe camera, the optical characteristic change indicative of patientmotion in the zone; compare the background optical characteristic of theframe to an aggregate background optical characteristic, the aggregatebackground optical characteristic based on the plurality of frames; andif the comparison determines that the background optical characteristicchanged relative to the aggregate background optical characteristic bymore than a predetermined amount, set the aggregate background opticalcharacteristic to the background optical characteristic of the frame anddampen optical characteristic of the plurality of frames in subsequentmonitoring for the optical characteristic change in the one or morezones. The term “luminance” can be replaced by “color” in the method ofFIG. 5 or another any embodiment. For example, various embodiments canconcern a system for monitoring a patient in a patient area having oneor more detection zones, the system comprising: a computing systemconfigured to receive a chronological series of frames and perform thefollowing steps based on the reception of each frame of thechronological series: calculate a background color of the frame; foreach zone of one or more zones of the frame, monitor for a color changeof the zone as compared to one or more frames of a plurality of framesthat were previously received from the camera, the color changeindicative of patient motion in the zone; compare the background colorof the frame to an aggregate background color, the aggregate backgroundcolor based on the plurality of frames; and if the comparison determinesthat the background color changed relative to the aggregate backgroundcolor by more than a predetermined amount, set the aggregate backgroundcolor to the background color of the frame and dampen color of theplurality of frames in subsequent monitoring for the color change in theone or more zones.

The techniques described in this disclosure, including those of FIGS.1-6 and those attributed to a monitoring system, a computing system, aprocessor, and/or control circuitry, and/or various constituentcomponents, may be implemented wholly or at least in part, in hardware,software, firmware or any combination thereof. A processor, as usedherein, refers to any number and/or combination of a microprocessor, adigital signal processor (DSP), an application specific integratedcircuit (ASIC), a field-programmable gate array (FPGA), microcontroller,discrete logic circuitry, processing chip, gate arrays, and/or any otherequivalent integrated or discrete logic circuitry. A “computing system”as used herein refers to at least one of the foregoing logic circuitryas a processor, alone or in combination with other circuitry, such asmemory or other physical medium for storing instructions, as needed tocarry about specified functions (e.g., processor and memory havingstored program instructions executable by the processor for detectingmotion in one or more zones, detecting environmental illuminationchanges based on background luminance, and performing various steps inresponse to one or both of detecting motion and detecting environmentalillumination change). The functions referenced herein and thosefunctions of FIGS. 1-6, may be embodied as firmware, hardware, softwareor any combination thereof as part of a computing system specificallyconfigured (e.g., with programming) to carry out those functions, suchas in means for performing the functions referenced herein. The stepsdescribed herein may be performed by a single processing component ormultiple processing components, the latter of which may be distributedamongst different coordinating devices. In this way, the computingsystem may be distributed between multiple devices, including part of acamera and part of a computer. In addition, any of the described units,modules, or components may be implemented together or separately asdiscrete but interoperable logic devices of a computing system.Depiction of different features as modules or units is intended tohighlight different functional aspects and does not necessarily implythat such modules or units must be realized by separate hardware orsoftware components and/or by a single device. Rather, functionalityassociated with one or more module or units, as part of a computingsystem, may be performed by separate hardware or software components, orintegrated within common or separate hardware or software components ofthe computing system.

When implemented in software, the functionality ascribed to a computingsystem may be embodied as instructions on a physically embodiedcomputer-readable medium such as RAM, ROM, NVRAM, EEPROM, FLASH memory,magnetic data storage media, optical data storage media, or the like,the medium being physically embodied in that it is not a carrier wave,as part of the computing system. The instructions may be executed tosupport one or more aspects of the functionality described in thisdisclosure.

The particular embodiments described below are not intended to limit thescope of the present disclosure as it may be practiced in a variety ofvariations and environments without departing from the scope and intentof the invention. Thus, the present disclosure is not intended to belimited to the embodiments shown, but is to be accorded the widest scopeconsistent with the principles and features described herein. Variousmodifications and additions can be made to the exemplary embodimentsdiscussed without departing from the scope of the present invention. Forexample, while the embodiments described above refer to particularfeatures, the scope of this invention also includes embodiments havingdifferent combinations of features and embodiments that do not includeall of the described features. Accordingly, the scope of the presentinvention is intended to embrace all such alternatives, modifications,and variations as fall within the scope of the claims, together with allequivalents thereof.

What is claimed is:
 1. A system for monitoring a patient in a patientarea having one or more detection zones, the system comprising: acomputing system that receives a chronological series of frames from acamera and performs the following steps based on the reception of eachframe of the chronological series: calculate a current backgroundluminance of a current frame; calculate an aggregate backgroundluminance based on a respective background luminance for each of aplurality of previous frames of the chronological series; for each ofone or more zones of the current frame, calculate a current number ofedges within the zone; for each of the one or more zones of the currentframe, detect patient motion based on a change between the currentnumber of edges within the zone and a previous number of edges within acorresponding zone from one or more previous frames; compare the currentbackground luminance of the current frame to an aggregate backgroundluminance; in response to the current background luminance changingrelative to the aggregate background luminance by more than apredetermined amount, disregard the detection of patient motion, set thecurrent number of edges for each of the one or more zones as theprevious number of edges for each of corresponding one or more zonesfrom the one or more previous frames, and set the current backgroundluminance as the aggregate background luminance in subsequent detectionfor patient motion; and in response to the current background luminancechanging relative to the aggregate background luminance by less than thepredetermined amount, generate an alert with a user interface based onthe detection of patient motion.
 2. The system of claim 1 furthercomprising the computing system configured to calculate the currentnumber of edges by analyzing contrast in luminance values betweenneighboring pixels within the zone.
 3. The system of claim 1 furthercomprising the computing system configured to detect patient motionaccording to distinctive rules for each of the one or more zones.
 4. Thesystem of claim 3 wherein the distinctive rules include a fall alert forgiven ones of the one or more zones corresponding to where a patient islikely to fall.
 5. The system of claim 3 wherein the distinctive rulesinclude disarming of patient motion detection in given ones of the oneor more zones corresponding to visitors.
 6. The system of claim 1wherein the predetermined amount comprises a dynamic threshold amountthat changes based on the current background luminance or a backgroundluminance of at least one of the plurality of previous frames.
 7. Thesystem of claim 1 wherein the predetermined amount comprises a dynamicthreshold amount that is smaller in darker environments and larger inbrighter environments.
 8. The system of claim 1 wherein thepredetermined amount comprises a percentage of a standard deviation ofthe current background luminance or a background luminance of at leastone of the plurality of previous frames.
 9. The system of claim 1wherein the predetermined amount is based on a relative amount ofinfrared content and non-infrared content of the current backgroundluminance.
 10. A system for monitoring a patient in a patient areahaving one or more detection zones, the system comprising: a computingsystem that receives a chronological series of frames and performs thefollowing steps based on the reception of each frame of thechronological series: calculate a current background luminance of acurrent frame; for each zone of one or more zones of the current frame,monitor for a change in a number of edges within the zone as compared toone or more frames of a plurality of frames that were previouslyreceived from a camera, the change in the number of edges indicative ofpatient motion in the zone of the current frame; compare the currentbackground luminance of the current frame to an aggregate backgroundluminance; and in response to the comparison determining that thecurrent background luminance changed relative to the aggregatebackground luminance by more than a predetermined amount, disregardchange in a number of edges as being indicative of patient motion in thecurrent frame, and monitor for changes in a number of edges for eachzone of one or more zones of a next frame as compared to the currentframe.
 11. The system of claim 10 further comprising the computingsystem configured to calculate the current number of edges by analyzingcontrast in luminance values between neighboring pixels within the zone.12. The system of claim 10 further comprising the computing systemconfigured to generate an alert in response to the monitoring stepidentifying the change in the number of edges.
 13. The system of claim10 further comprising the computing system configured to monitor for thezone luminance change according to distinctive rules for each of the oneor more zones.
 14. The system of claim 13 wherein the distinctive rulesinclude a fall alert for given ones of the one or more zonescorresponding to where a patient is likely to fall.
 15. The system ofclaim 13 wherein the distinctive rules include disarming of patientmotion detection in given ones of the one or more zones corresponding tovisitors.
 16. The system of claim 10 wherein the predetermined amountcomprises a dynamic threshold amount that changes based on the currentbackground luminance or a background luminance of at least one of theplurality of previous frames.
 17. The system of claim 10 wherein thepredetermined amount comprises a dynamic threshold amount that changesbased on the current background luminance or a background luminance ofat least one of the plurality of previous frames.
 18. The system ofclaim 10 wherein the predetermined amount comprises a dynamic thresholdamount that is smaller in darker environments and larger in brighterenvironments.
 19. The system of claim 10 wherein the predeterminedamount comprises a percentage of a standard deviation of the currentbackground luminance or a background luminance of at least one of theplurality of previous frames.
 20. A method for processing achronological series of frames generated by a camera to monitor apatient in a patient area having one or more detection zones byperforming the following steps based on the generation of each frame inthe chronological series, each step performed at least in part by acomputing system: calculating a current background luminance of acurrent frame; for each zone of one or more zones of the current frame,monitoring for a change in a number of edges within the zone as comparedto one or more frames of a plurality of frames that were previouslyreceived from a camera, the change in the number of edges indicative ofpatient motion in the zone of the current frame; comparing the currentbackground luminance of the current frame to an aggregate backgroundluminance; and in response to the comparison determining that thecurrent background luminance changed relative to the aggregatebackground luminance by more than a threshold amount, disregardingchange in a number of edges as being indicative of patient motion in thecurrent frame, and monitoring for changes in a number of edges for eachzone of one or more zones of a next frame as compared to the currentframe.